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This  paper  examines  the  problem  of  "positivity11  Tn  relation 
to  the  partial  realization  of  scalar  power  series.  An  exact  cri¬ 
terion  of  positivity  is  proved  for  second -order  realizations. 

The  general  case  is  currently  unsolved.  Even  the  special  results 
contained  here  show  that  the  so-called  "maximum  entropy  prin¬ 
ciple"  cannot  be  applied  to  the  realization  problem  in  the  naive 
sense  in  which  it  is  employed  by  physicists.  It  would  be  better 
to  call  this  principle  a  "prejudice"  because  it  does  not  fully 
utilize  the  information  inherent  in  the  data  and  does  not  pro¬ 
vide  a  realization  with  natural  ("minimal")  mathematical  pro¬ 
perties  . 


The  research  I  planned  to  report  on  is  unfortunately  not  yet 
completed.  So  the  .iollowing  is  only  an  outline  of  the  problem. 
It  has  been  around  for  a  long  time  without  receiving  a  defini¬ 
tive  solution.  It  occupies  quite  a  central  position  in  system 
theory  and  has  been  frequently  misinterpreted.  It  is  not  in  the 
least  controversial  but  it  is  unsolved. 

As  perhaps  the  only  algebraist  at  this  meeting,  it  is  safer 
for  me  if  I  use  a  reasonably  nontechnical  language.  In  such 
terms,  the  topic  of  my  paper  is:  What  is  the  relation  between 
partial  realizations  and  positivity? 

Evidently  I  must  now  define  "partial  realization"  and  (in 
relation  to  it)  "positivity". 

To  take  the  simplest  (scalar)  case,  consider  an  arbitrary 
sequence  of  numbers  from  a  fixed  field  k, 


*  Presented  at  Toeplitz  Memorial  Conference  at  Tel  Aviv  Univer¬ 
sity,  Tel  Aviv,  ISRAEL,  on  May  12,  1981.  This  research  was 
supported  in  part  under  US  Air  Force  Grant  AFOSR  76-3034 (® 
and  US  Army  Research  Grant  DAA  29-77-G-0225  through  the  Center 
for  Mathematical  System  Theory,  University  of  Florida. 
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(1)  3^  t  9  •  •  •  r  I  •••  • 

This  sequence  may  be  interpreted  as  an  element 

(2)  otz'1)  -  s5q  asz's 

of  the  ring  of  formal  power  series  z  ^k[[z  1  ]  ]  -  (We  shall  al¬ 
ways  adhere  to  the  normalization  convention  that  aQ  =  0.) 

Consider  now  any  (irreducible)  rational  fraction  written 
formally  as  it  (z) /\  (z)  ,  with  coefficients  in  k,  with 

=  monic,  and  with  deg  tt^  <  deg  xt-  Attached  to  each  such 
fraction  there  is  a  (unique)  formal  power  series  which  we  write  as 

(3)  VZ)/Xt<2>  “  s^o  V"S- 

(Another  normalization  convention  is  that  polynomials  are  written 
with  indeterminate  z  while  the  corresponding  power  series  are 
written  with  indeterminate  z  1.) 

Let  t  be  fixed  or  variable.  We  say  that  (it  ,  xfc)  is  a 
partial  realization  of  (1)  of  order  t  iff 

(4)  ag  =  bs>  s  =  1 ,  • • . f  t. 

There  is  a  rather  complete  theory  of  partial  realizations 
without  any  conditions  on  the  sequence  (1) .  An  elementary  ac¬ 
count  of  this  theory,  sufficient  for  the  present  purposes,  is  in 
KALMAN  [1979].  (A  complete  mathematical  treatment  will  appear 
as  KALMAN  [1983]  .) 

Assume  that  the  field  k  is  specialized  to  the  reals  R. 
Assume  also  that  the  sequence  (1)  is  replaced  (change  of  notation 
to  avoid  conceptual  confusion)  by  the  sequence 

(5)  cq,  c  ^  *  ^2 9  •••*  ^t*  *  *  * 
or,  equivalently, 

(6)  c  =  Z  c  z"S. 

s>o  s 

To  emphasize  the  conceptual  difference  between  (1)  and  (5)  we 
shall  normalize  the  latter  by  setting  c  =  1  (contrary  to 
ao*0). 
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We  want  to  regard  (5)  as  defined  via  the  covariance  function  of 
a  stationary  random  sequence  { }  with  zero  mean.  That  is, 

(7)  E(y  y  ,  )  =:  cov(y  y  ,  )  =:  c  =  c  , 

t  t+s  jt  i  +  s  s  -s 

s=0,  ±1,  ±2,  ...  . 

(By  stationarity ,  the  left-hand  side  of  (6)  is  independent  of 
x.)  Each  sequence  (5)  defines  a  sequence  of  Topelitz  matrices 


(8)  Tfc  := 


Directly  from  the  defining  properties  of  "covariance",  we  see 
that  (5)  is  a  "covariance  sequence",  that  is,  defined  by  (7), 
if  and  only  if  each  member  of  the  infinite  sequence 
Tq,  ...,  Tfc,  ...  is  positive  definite.  This  is  the  "positivity" 
property  of  the  sequence  (5)  which  we  wish  to  study  and  which 
justifies  the  normalization  c  =  1. 

In  other  words,  our  basic  problem  will  be:  What  happens  to 
the  theory  of  partial  realizations  if  the  sequence  ( 5 )  is  sub¬ 
jected  to  the  additional  condition  T  >  0? 

In  general,  questions  of  "positivity"  constitute  an  underde¬ 
veloped  area  of  mathematics.  Positivity  is  extremely  important 
for  system  theory  because  it  is  directly  related  to  questions  of 
stochastic  realization.  The  "identif lability"  of  noise  and  of 
stochastic  effects  comes  about  through  the  mathematical  implica¬ 
tions  of  positivity.  (For  some  preliminary  results  along  these 
lines,  see  KALMAN  [1982].)  Thus  each  theorem  about  positivity 
gives  rise  to  a  system-theoretic  result.  But  more  frequently,  a 
system-theoretic  question  poses  an  (open)  mathematical  problem. 
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The  classical  results  concerning  "positivity"  in  relation  to 
the  sequence  (5)  were  first  elucidated  in  the  study  of  orthogo¬ 
nal  polynomials.  Our  favorite  recent  reference  for  this 
material  is  GERONIMUS  [1961] . 

As  in  the  theory  of  orthogonal  polynomials,  it  is  useful  for 
us  to  introduce  the  sequence 


(9) 


r  —  (r  ^ ,  r  ^  >  •  • .  ,  r  ,  ...  ), 


sometimes  called  SCHUR  parameters,  defined  from  (5)  by 


(10)  r :=  (-  1 ) t-1 (det  Tt_1)"1det 


.  c 


...  c 


t-1  t 
t-2  ct-l 


°t-2  ct-3  ...  1 


For  any  positive  infinite  sequence  (5) ,  the  infinite  sequence  r 
is  well  defined.  The  classical  result  is  that  (5)  is  a  positive 
sequence  if  and  only  if  |r  |  <  1  for  all  t  =  1,  2,  ...  . 

By  elementary  arguments  based  on  (8)  and  (10)  it  can  be  shown 
that  there  is  a  bijective  correspondence,  for  each  t  >  0, 
between  positive  partial  sequences 
(1#  ,  *  ■  • '  ^  ^  { r*  i ,  ...,  rfc } . 

Any  partial  realization  problem  may  be  viewed  as  looking  for 
an  infinite  continuation  of  a  finite  sequence.  By  the  bijection 
between  c  and  r,  the  solution  of  the  positive  partial  reali¬ 
zation  problem  is  then  equivalent  to  finding  a  continuation  of 

(9)  by  rt+1*  rt+2'  *’■  SO  that  lrt+ul  <  1  for  311 
u  =  1,  2,  ...  .  It  is  not  clear,  however,  what  continuations  of 
r  correspond  (via  the  bijection)  to  rational  continuations  of 
c  as  required  in  the  definition  of  the  partial  realization 
problem. 

There  is  just  one  well-known  fact  concerning  such 
continuations.  The  special  continuation  given 
by  rg  =  0  for  all  s  =  t  +  1,  t  +  2,  ...  is  a  rational 
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positive  continuation  of  1,  c^,  ...»  cfc  and  corresponds  to  a 
solution  of  the  partial  realization  problem  with  deg  xfc  =  t. 

This  would  seem  to  provide  a  solution  to  our  problem.  In 

fact,  although  the  trick  described  above  is  mathematically 

quite  trivial,  the  realization  corresponding  to  rg  =  0,  s  >  t, 

has  been  glorified  by  physicists  by  naming  it  the  MAXIMUM 

ENTROPY  PRINCIPLE.  (See  LEVINE  and  TRIBUS  [1978]  for  recent 

speculations  and  references.)  The  terminology  is  due  to  the 

fact  that  continuation  of  (5)  with  r  =0  maximizes  the  clas- 

s 

sical  (SHANNON)  value  of  entropy.  See  (17-18)  and  below. 

In  mathematical  terms,  big  talk  like  "maximum  entropy" 
PRINCIPLE  would  seem  to  imply  that  the  solution  of  the  partial 
realization  problem  afforded  by  the  above  trick  has  some  "natu¬ 
ral"  attributes.  Unfortunately,  this  is  far  from  true.  There 
is  a  misunderstanding  on  the  part  of  the  physicists.  SHANNON 
entropy  does  not  represent  the  correct  "information"  measure  for 
the  realization  problem  discussed  here.  (Of  course,  SHANNON 
never  claimed  such  a  thing.  The  confusion  arises  from  identi¬ 
fying  "entropy"  as  technically  defined  by  SHANNON  with  the  word 
"information"  and  then  wildly  extrapolating  the  intuitive  mea¬ 
ning  of  the  latter.) 

That  the  conventional  application  of  the  "maximum  entropy" 
idea  is  incorrect  in  this  context  is  easily  seen  from  the  fact 
that  the  preceding  solution  of  the  positive  partial  realization 
problem  is  not  minimal,  in  the  sense  that  nfc  :=  deg  xt  is  not 
necessarily  minimal  among  the  family  of  all  (rational)  continua¬ 
tions  of  the  sequence  1,  c^,  ct<  Nonminimal  realizations 

are  objectionable  from  the  system-theoretic  viewpoint  as  they  do 
not  provide  the  "simplest"  explanation  of  the  given  data 
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To  make  these  statements  more  precise,  we  now  define  the 
polynomials 


(Ha) 


<J>t  (z) 


(det  Tfc_1) 


-1 


det 


C1  1 


't-1 


(*o(z)  =  1)  , 


and 

(lib)  4>*  ( z )  :=  zdeg  $t*$t(z_1). 

These  polynomials  arise  naturally  in  the  well-known  recursion 
relations  in  the  theory  of  orthogonal  polynomials,  namely 

(12a)  $t+l(z)  =  z,1>t(z)  ~  rt+l4>t<z)' 

(12b)  4>*+1(z)  =  "  rt+l$t(2)  +  $t{z)' 

Next,  we  need  a  little  algebra.  Let 


(13) 


(c,  TT)  ( - >  <C  ,  7T  >  :=  (CTT ) 


o-th  coefficient 

be  an  abstract  inner  product  between  power  series  and  polynomials 
the  product  ctt  is  to  be  taken  in  the  usual  sense.  Inserting 
(13)  into  the  determinant  defining  4>t  gives  <c,  4>fc>  =  0  for 
all  t.  More  generally, 


<c ,  z4>  > 

:  14)  r  =  - =  -  *.(0)> 

<c,  Vl> 


this  follows  immediately  by  applying  the  identity  <c,  4>t>  =  0 
to  (12a).  Formula  (14)  is  mildly  original. 

Now  it  is  easy  to  see  what  the  condition  rg  =  0,  s  >  t, 
amounts  to. 

By  formula  (14),  we  get 

(15)  <c,  z4>  >  =  0  for  s  >  t. 

s  ■= 

By  induction,  this  implies 
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(16)  <c,  zU4>t>  =  0  for  u  =  1,  2,  ...  . 

Using  also  (11a),  relations  (16)  express  the  fact  that  is  a 

recursion  polynomial  for  the  sequence  1,  c^ ,  ...,  c  extended 
to  an  infinite  sequence  by  the  assumption  rg  =  0,  s  >  t.  De¬ 
fining  xt  :=  4>t  gives  a  solution  of  the  positive  partial  reali¬ 
zation  problem. 

Note  that  0 1  is  a  recursion  polynomial  for  the  finite  se¬ 
quence  ct_i»  ct-2'  Co'  cl'  •••»  ct  f°r  eac^*  t. 

The  entropy  associated  with  a  covariance  of  length  t  is 
defined  in  the  sense  of  SHANNON  as 

(17)  :=  \  log  det  T  . 

* 

Note  that  det  T^/det  Tfc_^  =  <c ,  4»t>  by  (11a).  Then,  using 
(12b) ,  we  get  immediately 


(18) 


det  T  .  _  det  T 

- — — -  =  ( l  -  r  ) - — 

det  '  rt+l'det  T  . 

t  t-1 


This  shows  that  r  =0  for  s  >  t  is  indeed  that  continuation 

s 

of  r ^ ,  . . . ,  rfc  which  maximizes  SHANNON’S  entropy 
Ht+1'  Ht+2 '  "*  ’ 

Let  us  investigate  the  simplest  special  cases  of  this  re¬ 
sult  . 

Case  t  =  1.  Here  the  application  of  partial  realization 
theory  is  trivial:  all  minimal  partial  realizations  of  the  se¬ 
quence  1,  c^  are  given  by  the  fraction  z/(z  -  c^) .  From  for¬ 
mula  (11a)  we  see  that  =  (z  -  c^  is  identical  with 

^(z)  of  the  maximum  entropy  realization.  (This  is  just  a  lucky 
coincidence.)  The  positivity  condition  on  the  minimal  partial 
realization  is  |r^|  =  |c^|  <  1.  Since  the  parameter  c^  is 
"free"  in  the  minimal  partial  realization  problem,  here  the  po¬ 
sitivity  requirement  does  not  interfere  with  the  minimality  re¬ 
quirement.  (In  the  fraction  for  the  partial  realization  the  fac¬ 
tor  z  is  due  to  the  different  normalization  of  (5)  vs.  (1).) 
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It  would  be  a  great  mistake,  however,  to  imagine  that  this 

is  typical  of  the  general  situation. 

Case  t  =  3.  Here  we  study  positive  minimal  partial  reali- 

2 

zations  of  the  partial  sequence  1,  c^,  c^,  c^.  If  c2  ^  ci 

then  the  minimal  realization  problem  has  a  "jump"  of  size  1  at 

t  =  2  (see  KALMAN  [1979]),  c^  is  a  free  parameter,  and 

nmin  _  (j£  _  c2  jjUt  ^  then  the  minimal  partial 

realization  problem  has  a  jump  of  size  2  at  t  =  3,  so  that  now 

c^  and  c,.  are  free  parameters,  and  n™in  =  3.  This  case  is 

already  rather  difficult  to  analyze.) 

2 

We  assume  c 2  ^  c^  which  is  equivalent  to  r2  f  0.  Then 
the  minimal  partial  realization  is  given  by  the  finite  continued 
fraction 


(20)  |rj  <  1,  |  r2 1  <  1,  |r3j  <  1 

are  obviously  necessary  for  the  partial  sequence  1,  c^,  c2 ,  c^ 
to  be  positive,  irrespective  of  the  assumed  jump  pattern.  These 
positivity  conditions  can  be  easily  satisfied  without  conflict¬ 
ing  with  the  minimality  requirements  of  the  partial  realization; 
r^  and  r^  are  free  parameters  and  the  condition  r2  f  0  does 
not  interact  with  ]r2|  <  1. 

But  (20)  is  not  sufficient  to  insure  that  the  infinite  se¬ 
quence  generated  by  (19)  is  positive!  The  necessary  and  suffi¬ 
cient  condition  for  the  latter  requirement  is  (after  rather  ex¬ 
tensive  calculations)  found  to  be 
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This  is  much  stronger  than  |  r  ^  |  <  1.  Thus,  the  positivity  con¬ 
ditions  on  a  partial  sequence  are  not  sufficient  to  guarantee 
that  the  corresponding  minimal  partial  realizations  generate  a 
positive  (infinite)  sequence . 

When  r^  does  not  satisfy  (21)  ,  the  minimal  partial  reali- 

J  2 
zation  of  1,  c^,  c (assuming  c2  /  c^)  is  not  positive? 

conversely,  the  minimal  positive  partial  realization  of  this  se¬ 
quence  is  then  at  least  of  degree  3  (but  not  higher,  because  of 
the  existence  of  the  maximum  entropy  realization) . 

The  preceding  investigation,  which  is  of  course  mathemati¬ 
cally  rigorous,  shows  that  for  t  =  3  the  maximum  entropy  reali¬ 
zation  is  not  necessarily  a  "natural"  realization  since  there 
may  exist  a  minimal  realization  of  degree  2.  (If  1,  c, ,  c2, 
corresponds  to  r^,  0,  0  then  4> ^  ( z )  =  z2<t>1(z)  so  that  the 
maximum  entropy  principle  does  provide  the  unique  minimal  reali¬ 
zation  of  degree  1.  But  this  is  just  another  lucky  accident.) 

If  it  is  true  that  the  maximum  entropy  principle  fails  to 
reliably  select  the  simplest  (here  "minimal")  realization,  then 
this  alleged  organizing  principle  of  nature  must  contain  some 
extraneous  assumptions  (which  I  propose  to  designate  by  the 
technical  term  prejudice ,  as  in  KALMAN  [1982]).  What  are  these 
extraneous  assumptions?  Where  does  the  prejudice  come  in? 

JAYNES  [1968]  describes  the  principle  as  follows: 

"The  [prior  probability  assignment]  that  describes 
the  available  information  but  is  maximally  noncom¬ 
mittal  with  regard  to  the  unavailable  information 
is  the  one  with  maximum  entropy  (my  italics)." 

It  is  hard  to  quarrel  with  this  statement  on  intuitive  grounds. 
Everything  hinges  on  the  meaning  given  to  the  word  "information". 
JAYNES  and  his  followers  apparently  blindly  accept  that  SHANNON 
entropy  =  information.  But  entropy  is  never  a  measure  of  "avail¬ 
able  information"  of  the  mathematical  type. 
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In  the  literature  (see,  for  example,  ULRYCH  and  BISHOP 
[1975])  the  maximum-entropy  realization  is  regarded  as  equivalent 
to  estimation  by  the  autoregressive  (AR)  method.  (This  is  in 
fact  easily  proved  with  the  help  of  the  machinery  developed 
here.)  Thus  "maximum  entropy"  becomes  a  "higher"  justification 
for  the  AR  method  or  vice  versa.  But  the  AR  method  is  also  ob¬ 
jectionable;  it  implies  the  tacit  assumption  (prejudice)  that  the 

partial  realization  problem  consists  of  one  concentrated  jump - 

which  is  a  highly  nongeneric  case. 

The  problem  of  attempting  to  apply  the  maximum  entropy  dic¬ 
tum  of  JAYNES  via  formulas  (17-18)  is  simply  that  the  SHANNON 
entropy  does  not  correctly  measure  the  "available  information" 
provided  by  values  of  r ^ ,  ...,  rfc.  This  is  not  a  question  of 
probabilities;  the  computation  of  4>t(z)  is  based  on  the  exact 
knowledge  of  r^,  ...,  rfc. 

Consider  again  (19).  If  r2  ^  0  but  | r 2 1  is  small  then 
by  (20)  | r 3 1  must  be  small  also  and  therefore  ,  the  de¬ 

nominator  of  (19)  viewed  as  an  ordinary  fraction,  is  very  close 
to  ^(z)  =  z  -  r^.  This  means  that  the  correct  (minimal)  solu¬ 
tion  of  the  partial  realization  problem  is  very  similar  to  the 
case  where  the  only  data  is  r ^ ,  even  though  r2  and  r^  are 
not  exactly  zero. 

On  the  other  hand,  if  |r2|  is  nearly  1  then  X2^z^  will 
differ  from  4>2(z)  only  by  a  term  bounded  by  1  -  |  r 2 1  while 
| r^ | ,  far  from  being  required  to  be  zero  as  by  the  maximum 
entropy  method,  can  be  almost  as  large  as  1/2. 


We  see  that  the  realizations  provided  by  the  maximum  entropy 
prejudice  are  approximately  valid  in  a  much  wider  range  then  the 
rigid  assumption  that  |r  |  =  0  for  s  >  t.  To  see  whether 
(z) >  t^  c  t  is  approximately  valid  for  the  sequence 
1,  c,  ,  . . . ,  c.  ,  . . . ,  c  we  need  certain  bounds  on  I  r,_  .1, 

1  ti  t  1  ti+1 


t^  <  t  is  approximately  valid  for  the  sequence 

. .  ,  c.  ,  . . . ,  c  we  need  certain  bounds  on  I  r,_  , .  I  , 
t^  t  1  tj+11 

...  which  guarantee  that  the  infinite  continuation  of 


I  *  +  2  I  9  •••  wiuwu  yuaxaiiuce  unat  uiic  uiiujike 

the  sequence  by  a  partial  realization  of  order  t^  is  still 


341 


positive.  It  is  these  missing  bounds  which  constitute  the  un¬ 
solved  mathematical  problem.  Results  of  the  writer  at  present 
are  limited  to  (19)  and  its  corresponding  bound  (21)  .  In  gene¬ 
ral,  if  I  rs  I  is  sufficiently  small  for  t^  <  s  £  t,  we  may 
expect  to  get  positive  partial  realizations  that  are  much  more 
efficient  than  the  maximum  entropy  realization.  Thus,  to  cor¬ 
rectly  implement  JAYNES'  intuitive  principle,  it  is  necessary 
to  know  the  values  of  the  lrsl*  0n  t^e  other  hand,  the  con¬ 
ventional  application  of  the  maximum  entropy  principle  (desig¬ 
nating  <t>t(z)  as  the  recursion  polynomial)  uses  the  classical 
entropy  formula  (17)  which  requires  only  the  "information"  that 
I r  [  <  1  for  1  <  s  <  t;  the  actual  values  of  r  are  used 
in  constructing  $t(z)  but  not  in  applying  (17). 

Thus  the  prescription  rg  =  0,  s  >  t,  should  be  regarded 
only  as  a  realization  of  "last  resort",  precisely  because  in 
doing  so  we  have  not  exploited  all  available  information. 

As  soon  as  the  mathematical  problem  posed  here  is  solved,  we 
would  be  in  a  position  to  implement  JAYNES'  desideratum  of  not 
throwing  away  any  information  concerning  the  sequence 
r^,  ...,  rfc.  By  working  backward,  we  would  then  replace  formula 
(17)  by  a  different  expression  which  would  correctly  measure 
"information"  as  it  is  relevant  to  the  realization  problem. 
Evidently  the  correct  measure  of  "information"  for  a  mathemati¬ 
cal  problem  will  not  be  the  same  as  the  measure  of  entropy  for 
a  physical  problem. 

"Entropy"  is  the  measure  of  the  lack  of  organization 
(structure)  in  a  physical  situation?  it  has  nothing  to  do  with 
the  "system  aspect"  of  things  and  therefore  entropy  is  useless 
not  only  as  a  technical  tool  but  also  as  a  conceptual  crutch. 
Were  Nature  so  constituted  that  simple  principles  explain  every¬ 
thing,  "maximum  entropy"  would  be  one  of  the  deep  insights  and 
we  would  not  need  Mathematics.  But  Nature,  if  sometimes  simple, 
is  often  complicated  and  therefore  Mathematics  is  indispensable. 
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